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METHOP, APPARATUS, AND SYSTEM FOR SYNTHESIZING AN AUDIO 
PERFORMANCE USING CONVOLUTION AT MULTIPLE SAMPLE RATES 

CROSS REFERENCE ipO RELATED APPLICATIONS 

This application ctoims the benefit of US provisional patent application nos, 60/510,068 and 
60/510,019, both filed on October 9, 2003» 

BACKGROUND OF THE INVENTION 

CQMPLfTER LISTING APPENDIX 

This application includes a Computer Listing Appendix on compact disc, hereby incorporated by 
reference. 

1- p^ejd of the Invention 

[0001] The present invention relates generally to andio processing and, more particularly, to a 
method, apparams, and system for synthesizing an audio performance in which one or more 
acoustic characteristics, such as acoustic space, microphone modeling and placement, are varied 
using pseudo-convolution processing techniques. 
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2. Description of the Prior Art 

[0002J Digital music syartiesiz^rs are known in the art. An example of such a (Jigital music 
synthesizer is disclosed in U.S, Pateni No, 5, 502, 747, hereby incorporated by reference. The 
system disclosed in the '747 patent discloses multiple component filters and is based on hybrid 
time donuun and &equeju:y domain processing. Unfortunately, the methodology utilized in the 
5,502,747 patent is relatively computationally intensive and is thus tiot efficient. As such, the 
system disclosed in the *747 patem is primarily only useful in academic and scientific 
^plications where computation time is not critical. Thus, there is a need for m efficient 
synthesizar that is relatively more o^ficient than those in the prior art. 




SUMftfAfiY OP THE INV^JTyiON 
[0003] The present invention relates to a method, apparams, and system for use in synthesizing 
an audio performance in which one or more acoustic characteristics, sueh as acoustic space, 
microphone modeling and placement, can selectively be varied* In order to reduce processing 
time, the system utilizes pseudo-convolution processing techniques at a greatly reduced 
processor load. The system is able to emulate the audio ou^ut in diiferent acoustic spaces, 
separate musical sources (itistruments and other sound sotirces) &om musical context; 
interactively recombine nmsical source and miisical context with relatively accurate acoustical 
integrity, including surround sound contexts, emulate nticrophone models and microphone 
placement, create acoustic effects, such as reverberation, emulate instrument body resonance and 
interactively switch emulated insnument bodies on a given nmsical instrument. 
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Bfil^F DESCRIPTION OF THE DRAWING 

10004] FIGS- JA and IB ai^ exemplary graphical user inierfiices for use with ibe present 
mvention. 



lOOOS] flGS. IC and ID are alternate exemplary graphical user interfaces for use with the 
preset inventioQ. 




10006] FIG, 2 is a high level block diagram of cme erobodiroent of the present invention; 



[0007] FIG, 3 is a block diagram of an exemplary embodiment of a nm-time input channel 
processing routine designated by the block 50 in Fig- 2 in accordance with the present invention; 



10008] no. 4 is a more detailed block diagram of the embodiment illustrated in FIG. 2; 




(0009] FIG. 5 is a block diagram illustrating a process channel mutine designated by the block 
S3 in Fig. 2 in accordance with the present invention; 



[OOlOJ FIG. 6 is a time domain response of an exemplary sound impulse; 



1001 IJ FIG, 7 is a block diagram of an audio collection and index sequencing routine illustrated 
by the block 178 in Fig. 5 xa accordance with the present invention, represented by the blocka 
178a, I78b and 178c-, which illustrate different operational modes for the Audio Collection and 



Index Servicing Routine in accordance with the the present invention 
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10012 J FIG, 8 is a block diagram of coefficient index sequencmg routine illustrated by the block 
1 70 in Fig. 5 in accord wee with the present invention; and 



[001 3J FIG. 9 is a block diagram of the collection index modulo update routine illustrated by the 



block 192 in Fig, 7 in accordance with the present invention. 

10014J FIG. 10 is a block diagram of the frame modulo update in accordance with the present 
invention; 



lOOlS] FIG. U is an «emplary block diagram of the tail extension processing in accordance 
with the present invention; 

[0016] FIG. 12 is a hardware block diagram of a computing platform for use with the present 
invention. 



jAn . li^P DESCRIPTION 

[0017J The present invention relates to an audio processing system for synthesizing an acoustic 



audio response in a selectable musical context or acoustical space can be emulated. In particular, 

» 

a model of virtually any acoustic space, for example, Carnegie Hall, can be recorded and stored. 
In accordance with one aspect of the invention, the system emulates the acoustic response in the 
selected acoustic space model, such that the audio input sounds as if it were played in Carnegie 
Hall, for example- 
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[0018] Jn accordance with one aspect of the invention, the system has the ability to separate 
musical sources (i.e. instruments and other sound sources) £rom the musical context (i.e, acou$tic 
space in which the sound sources are played). By emulating the response to selectable music 
contexts, as described above, the acoustic response to various musical sources can be emulated 
fbr virmaJly any acoustic spac^^ including the back seat of a station wagon. 





[0019] Various techxuques can be used for generating a model of an acoustic space. The model 
may be considered a fingerprint of a room or other space or musical context. The model is 
created, for example, by recording the room response to a sound impulse, such as a shot from a 
staner pistol or other acoustic input. The sound impulse maybe created, for example, by placing 
a speaker in the loom or space to be modeled and playing a tequeqcy sweep. More particularly, 
a common technique is the sine sweep method which has a sweep tone and a complementary 
decode tone. The convolution of the sweep tone and the decode tone is a perfect single sample 
spike (impulse). After the sweep tone is played through the speaker and recorded by a 
microphone in the loom^ the resulting recording is convolved with the decode tone which reveals 
the room impulse response. Alternatively, simply firing a starter pistol in the space and reclrding 
the response is another way. Alternatively, various •*canned'' acoustic space models are currently 
available on the Internet at hti|>:/www,ecbochamber-ch[?]; bttp:/alliverb.daw-mac,com; and 
httpi/poisevaultcom . 



(0020] In accordance with other asfpects of the invention, the system is able to emulate other 
acoustic characteristics, such as the response to one or more predetemuned microphones, such 
as a vintage AJCG C-12 microphone. The microphone is emulated in the same manner as the 
musical context. In particular, the acoustic response to an acoustic impulse of the vintage 
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microphone, for example, is reconied and stored. Any musical source played through the system 
is processed so that it sounds as if it were played through the viniage microphone. 





10021J The system is also able to emulate other acoustic characteristics, such as the location of 
an audio source within an audio context. In particular, in accordance with another aspect of the 
invention, the system is able to combine a soimd source, the response of an acoustic space, a 
mictophone and an instrument body resonance response into separate, reconfigurable audio 
SQWces in an audio performance. For example, when an instrument, say a violin, is performed 
in a room and recorded thtough a microphone, ihe resulting audio contains tonality and 
reverberation dictated by multiple impulse elements, namely the microphone, room acoustics and 
the violin body. In many cases it is desirable to control these three elements individually and 
separate from each other and th^ sning vibration of ihe violin. By doing so, different choices of 
microphone, room environment or violin body can be ind^endenily selected by the user or 
content author for an audio perfotmance. In additiot^ the system is able to optionally emulate 
the response to another audio characteristic, such as the location of an audio source relative to 
the microphone placement, thus allowing the audio souice to be vinuaUy moved relative to the 
microphone. As such, drums, for example, can be made to sound closer or further ^art from the 
microphones. 



10022J In accordance wi^ another aspect of the invention* the system is a real lime audio 
pnDcessing system that is significantiy less computation intensive than known music 
synthesizers, such as the audio processing system disclosed in the 747 patent discussed above. 
In particular, various techniques are used to reduce the processing load relative to known 
svstmis. For example, as will be described in more detail below, in a *Turbo" mode of 
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opemtion, rtie system processes iwput audio samples at a slower sample rate than the input 
sample me thus reducing the processor load up to 75 %, for e;carople. 



An exemplary host computing platform for use with the present invention is illustrated in FIG. 




12 and generally identified with the reference numeral 20. The host computing plaifom:i when 
loaded with the usar interface and processing algorithm described below fonns an audio 
synthesizer. The host computing platform 20 includes a C5*U 22, a random access memory 
(RAM) 24, a hard drive 26 , as well as an external display 28, m external microphone 30 and one 
or more external speakers 32. Minimum requirements far the host computing platform 20 are ; 
Windows XP (Pro, Home edition, embedded or other compatible operating system), an Intel 
Pentium 4, Celeron. Albion XP I GHz or other CPU, 256MB RAM, 20GB bard drive. 



USER INTERFACE 




[00231 FIGS- lA illustrate graphical representations of exemplary embodiments of a 
control panel 100 which may be used in connection with Ibe present invention. Only one 
embodunent is described for simplicity. In particular, in the embodiment illustrated in FIG- 1 A, 
the control panel 100 ixKludes a drop-down menu 102 which may be used to select a 
predetermined musical context C^.^., dark, hardwood floors, medium-..). « drop-down menu 104 
which may be used to select a 'Iraw impulse", a drop-down menu 106 which may be used to 
select a paiticular musical instrument violins, Legato down bows), a drop-down menu 

108 which may be used to select an original microphone (e-g., NTIOOO), and a drop-down menu 
110 which may be used to select a particular replacement microphone (eg., AKG414). A 
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cUsplay area U2 is provided for displaying a brief textual description of a microphone placement 
selection, as described in more detail below. 




(00241 A button 114 ia provided for selectively enabling and disabling a "cascade'* feature 
associated witb application of the raw impulse selected via tlte diT0p'4own menu 104 to an audio 
track. A button U6 is provided for selectively wabling and disabling an '*enco4e" featwe wbicli 
pennits tbe application of a user-selected acoustic model to the instrument selected via the drop- 
down menu 106, A display area 118 optionally may show a graphical or photographic 
representation of the musical context selected by the drop-down m»iu 102. 



1002S1 A button 120 is provided for selectively activating and deactivating a mid/side (M/S) 
microphone pair arrangement for left-side and right-side microphones. Additional buttons 121, 
122, 123, and 124 are provided for specifying groups of microphones, including, for example, all 
microphones (button 121), front C*F") microphones (button 122X wide C^W*") microphones 
(button 123), and rear or surround ("S") microphones (button 124)- 



10026] The user also may emer microphone polar patterns and roll*off characteristics for each of 
the microphones employed in any given simulation. For that purpose, buttons 124, 125, 126, 
127, 128, and 129 are provided for selecting a microphone roU-off characteristic or response. 
For example, buttons 125 and 126 select two different low-frequency bumps; bunon 127 selects 
a flat response, and buttons 128 and 129 select two different low-frequency roll-off responses, 
respectively. Similarly, buttons 130-134 allow a user to select one of several different well- 
recognized microphone polar patterns, such as an omni-direcdonal pattem (button 130), a wide- 
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angle cardioid panero (bution 131), a cardioid panera (button 132), a hyper cardioid pattern 
(buwon 133), or a so-called '*figure-8" pattern (button 134). 





{0027} The control panel 100 also includes a placement control section 135, wbicb, in tbe 
illustrated embodiment, contains a plurality of placement selectgr/indicator buttons (designated 
by numbers 1 tbiough 1 8), These placement selector/indicator buttons allow a user to specify a 
position of musical instruments within the use^selected musical context (e.^., the position of the 
instrument selected by the drop-down menu 106 relative to the user-ispecified microphone(s))- 
The grw)u<^al display area 1 18 may display a depiction of tho perspective of the room or musical 
context selected by ttie drop-down menu 102 corresponding to the placement within that room or 
musical context specified by the particular placement selector/indicdtor button actuated by the 
user. Of course, as will be readily apparent to those of ordinary slall in the an, many different 
alternative means may be employed to permit a user to select instrument placements within a 
particular musical context in addition to or instead of the placement selector/indicator buttons 
shown in FIG. i A. For example^ a graphical depiction of the rpom or musical context could be 
di^layed, and a mouse, trackball, or other conventional pqinter control device could be used to 
move a location designator to a predetermined placement within the graphical depiction of the 
room or musical context corresponding to whatever placement within that room or musical 
context may be desired by the user- 



[0028] As also shown in FIG. lA, the control panel 100 also includes a •"mic-to-ouipuf ' conttol 
section 136, which includes an array of bunons allowing a user to assign each microphone used 
in a given simulation to a corresponding mixer output channel As shown, the control panel 100 
provides for seven mixer output channels represented by the columns of buttons numbered one 
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through seven in the mic-to-output comrol section 136. Seven mixer output channels allow for 
seven micropbones to be used in a given simulation {e.g., leit and right front, left and right wide, 
left and right surround, and a center charmel). Of course, those of ordinary skill in the art will 
readily predate that more or fewer mixer output channels may be provided in any given 
embodiment of the present invention based upon the needs of a particular simulator. For 
example, in a stereo simulator, only two mixer output cbamiels need be provided. In order to 
assign a particular microphone to a particular mixer output channel, the user need only depress 
the button in the row of buttons corresponding to the particular microphone and the column of 
buttons corresponHing to thfi particular mixer output chamieL The controls in each row of the 
mic-to-output control section 136 operate in a mutually ^elusive ^shion, such that a particular 
microphone can be associated only with one mixer output channel at a time. 




[0029] The mic-to<)utput contml section 136 also includes a button 140 for selectively enabling 
aiul disabling a ''simulated stereo'' mode in which a single microphone simulation or output is 
processed to develop two 0.^., stereo) mixer output channels. This may be used, for example, to 
enable a simulated stereo output to be produced by a alow computer whicb does not have 
sufficient processing power to handle full stereo real-time processing. A button 142 is provided 
for selectively etiabling a ^true stereo'' mode, which simply couples left and right stereo 
microphone simulations or outputs to two mixer output channels. Further, a button 144 is 
provided for selectively enabling and disabling a '^seven-channel" mode in which each of seven 
microphone simulatioi^ or outputs is coupled to a respective mixer output channel to provide for 
ftill seven-channel surround sound output. 
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10030] A button 146 is provided for selectively enabling aiuj disabling a 'lail extend*' featuie 
which causes the illusi^ted synthesizer to derive liie first N seconds of the symbesized response 
by perfonning a full convolution and then to derive an approximation of the tail or terminal 
portion of the syntttesized response using a recursive algoriibra (described in more detail bejow) 
which is lossy but computationally efficient Where exact acoustically simulation is not 
required, enabling the tail extend feature provides a trade-off between exact acoustical 
simulation and computational overhead. Associated with the tail extend feature are three 
parameters, Overlap, Level, and Cutofi; and a respective slider conurol 148, 150, and 152 is 
provided for a4iu&tment of each of these parameters. 




10031] More particularly, the sUder control 148 pemiits adjustment of an amoum of overlap 
between the lecursively generated tail portion of the synthesi^sed response or output signal and a 
time-wise prior portion of the output signal which is calculated by convolution at a particular 
sample rate. The slider control 150 permits adjustment of the level of the recursively generated 
portion of the output signal so that it more closely matches the level of the time-wise prior 
convolved portion of the output signal. The slider control 152 pennits adjustment of the 
ftequency-domain cutoff between the recursively generated portion of the output signal and the 
time-wise prior convolved portion thereof to thereby smooth flw overall spectral damping of the 
synthesized response or output signal such, that the frequency-domain bandwidth of the 
recursively generated portion of the output signal more closely matches the frequency domain 
bandwidth of the convolved portion thereof at the nransition point betwew those two portions. 



(0032) A plurality of fiUTher slider comrols may be provided to allow a user to adjust the level 
corresponding to each microphone used in a particular simulation. In the illustrated 
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embodimeni, slider coiurols 154-160 provided for adjusting recordiug levels of each of seven 
recording channels, each corresponding to one of the available microphones in ibe iUnsnnaied 
simularion or synthesizer system. In addition, a roaster slider control 161 is provided to allow a 
i^ser to simultaneously adjust the levels set by each of the slider controls 1 54-1 60. As shown, a 
digital read-out is provided in landrail with each slider control 154-161 to indicate numerically to 
the user the level set at any given time by the corresponding slider control 154-161. In the 
illustrated embodiment, the levels are represented by 11 -bit niunbers ranging from 0 to 2047- 
However, it should be evident to diose of ordinary skill in the art that any other suitable range of 
levels in any suitable units could be used instead. 




|00331 The control panel 100 also includes a level button 164, a perspective button 166, and a 
pre-delay bunon 168. The level bunon 164 allows a user to selectively activate and deactivate 
the level controls 154-161 . The perspective button 166 allows the usct to selectively activate apd 
deactivate a perspective feature which allows the slider controls 154-161 to be used to a4)ust a 
parameter which simulates, for any given simulation, varying the physical dimensions of the 
musical context or room selected by the drop-down menu 102. The pre-delay button 168 allows 
the user to employ (he slider connt)ls 154-161 to a^ust a parameter which simulates echo 

y 

response speed (by a4}ustiQg the simulated lag between the initial echo in a recorded signal and a 
predetermined amount of echo density buildup). 



{0034] Alternate exemplary ^aphical user interlaces (GUQ are illustrated in FIGS. 16-lD. 
These GUIs also permit a user to adjust the various parameters of the system in accordance with 
the principles of ^ present invention. Since the GUI provides essentially the same fimctionality 
as the control panel illustrated in FIG. I A* the alternate GUIs are not described further. 
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PROCESSING ALGORITHM 



[0035] FIG, 2 depicts a high-level software block diagram* iUustnrting a single audio channel %ox 
sinr^liciiy, of aa excn^plary emhodimem of m audio processing system 48 in accordance with 
ihe present invention. The audio processing system 4S includes a runtime input c hanne l 
processing routine 50. a runtime sequencing, control, and data manager 52, a process-channel 
module 53 which includes a multi-rate adaptive filter 54, a collection and alignment routine 56, 
md a tail extension processor 58. As shown, irqiut digital audio source samples are digitized by 
3n analog to digital converter (not shown), for example, a 16 bit or 24 bit, PCM. 44.1, 48, 88.2, 
96, 176.4 or l92kH2 sample rate, mono or mulii channel AOC, such as the stereo ApC within 
the CizTus Crystal CS4226 codec, and ^plied to the runtime input channel processing routine 50, 
which converts the sample, which are in the time domain to the frequency domain and appUes 
the frequency domain samples to the runtime seqxiencing, control and data manager 52. In 
addition, impulse response data representing, for example, the impulse responses corresponding 
to the characteristics of various audio characteristics, such as, user-selected microphones, 
musical context(i.e. acoustic space), mvisical instruments, and relative positioning of the user 
selected mierophoncs and/or mttsical instruments within the user selected musical context are 
stored in a coefficient storage memory device 60. A loadiime coefiBcieni processing routine 62 
and a runtime coe£&cient processing routine 64 are used to successively process coefficients 
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from the coefficient memory storage device 60 based cm the usw input 66 provided via, for 
example, a control panel or graphical user interface, such as depicted in FIGS. I A -ID 




I0036J , In order to reduce matime CPU resource miUzaiioiw tJ^ loadtime coefliciem processing 
routine 62 pre-processes at load tinie *e time domain impulse coefficients from siorage device 
60 with audio signal processing to facitiiate changes to the audio response based on user input, 
and converts the resulting time domain coefficient data into the frequency domain- The runtime 
sequencing, control, and data manager 52 processes the audio source jj^ut samples and the 
processed impulse response coefficients as to facilitate CPU load balancing and efficient real 
lime piocessing. The processed samples and coefficients from the runtime sequencing, control, 
and data manager 52 are applied to the process channel module S3 in order to produce audio 
output samples 68 , which emulate the audio response of tbe mpul audio source to various user 
selected audio characteristics. 




[0037] FIG. 3 illustrates a block diagram of one exemplary embodiment of the runtime input 
channel pn>cessing routine 50 shown in FIG. 2. Referring to FIG. 3, the runtime input channel 
processing routine 50 receives digitized audio souxi^ samples at a first sample rate, for example 
4SkH2, from a digital sample buffer (lOBUF) 70. The digital sample buffer 70 i$ sized 32 audio 
samples of 32bits each. Digital samples from the digital sample buffer 70 are copied on a frame- 
by^frame basis by frame copy routines O) and (A) 72 and 74> respectively, to respective frame 
buffers (XLB) and (Xl-A) 76 and 78, respectively. More panfcularlyt the same input samples are 
framed into two separate buffers, XLB and XLA, of potentially dif&rem frame si^es as to 
facilitate subsequent processing at two different sample rates- The frame size of the XLB buffer 
is smaller relative to XLA, typically one eighth in size relative to XLB. The tail maintenance 
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loutine 80 copies a fmiie impulse response (FIK) fiUer lengtti of data from the begiwwig to the 



J of the fi^e buffei XLA, as to cover the FIJI coefificiem overlap requirea by the 2:1 



decimation filter 90. The decimation filter 90 downsample^ the wwe XLA fnme size of audio 
source samples, said frame size coiresponding to the lower sample rate, for example Vi the audio 
source sample rate md copies these samples to the decimation firame buffer (Xl.lp) 92. 





[Q038] A fast Fourier transform (FFD module 82, including FFT routines 84, 86, and 88, is 
provided for converting frames of data, which arc represented in the time domain in frame 
buyers 76 and 78 into coiresponding frequency-domain data* More particularly, the FFT routine 
84 produces a fast Fourier transforai of an XLB frame from the frame buffer 76 and provides the 
transformed data to a frequency domain buffer (XLBF) 94* In a turbo mode, friame data from the 
frame buffer (XLA) 78 is filtered by a low-pass filter, fi>r example a 2:1 filter to reduce the 
sample rate to 14 of the audio input source sample rate. The low pass filter simply reduces the 
audio bandwidth to one half of the input sample bandwidth and truncates the result by saving 
only every other sample. The filtered samples are stored in a decimation frame bufifer (Xl-lP) 92. 
This decimation frame bu^^ 92 contains the band reduced and truncated samples produced by 
low pass filtering and throwing away every other sample, and passes these samples to the FFT 
routine 86 which performs an FFT on the decimated, filtered frame data and stores the resulting 
frequency domain frame data in a frequency domain buffer (XI-AF) 96, 



[0039] In the event a user wishes not to employ tail end processing ite., preferring instead to 
achieve the acoustic accuracy of fiill-sannile-rate convolution which results greater processing 
power), the FFT module 88 may be operated at the full sample rate (i-e, same sample rate as the 
input samples) to uransfomt the frame data from the frame bu£fer (Xf^) 78 at its original sample 
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rate atui thus provide full-sample-r«e frequency domain data to the frequency domain buffer 96 
(XLAF). 

[0040] Operation of tbe frame copy routines (B) and (A) 72 and 74, The tail maintenance routine 
80, The FFT modtde 82, and the low-pass filter 90 is handled by a frame contxol process routine 
98. The frame control process routine synchronizes the timing of the frames so that they wotk in 
phase together, assembling a frequency domain frame which is larger than the time domain 



frames. The frame control process also synchronizes the multiple sample rates atid frame sizes 
of the XLA, XLB, XLAF> and XLBF buffers, as fed into die real time scheduling and CPU load 
balancing routines within the runtime sequencing, control and data manager 52. 

FIG 4 depicts a block diagram illustrating in greater detail the audio processing system shown in 
FIG 2, including an expanded illustration of the flow of data that occurs hi operation of that 
system. As shown, a plurality of audio soume input channels are shown, CH. 1, CH, N. 
As discussed above, the audio source h^ut channels CH. 1, CH. 2, . ..CH. N. are each process 



by the runtime input channel processing routine SO (FIG. 3) which is used to convert the time 
domain audio source samples, segregated mto multiple sample rates, to their respective 
frequency domain buffers for fintber processing. As discussed above, the frequency domain 
samples for each channel are stored in a plurality of frame buffers XLBfl, XLA£2..; XLAfN, 
identified with the reference numerals 102, 103 and 104, re^ectively, one fime buffer for each 
channel. Each of the frame buffers 102, 103. 104 is sized to receive one frame of input audio 
samples at a time from a corresponding one of the N audio input channels, for example 2048 
32bit samples. The run time memory JOO also includes a plurality of data structures 106, 107, 




frame size, such that an entire frequency domain frame is made up of multiple time domain 
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108 which represent die coefficienTs, for example, of M impulse rcspoijses for M ?icoustic 
cbamcteristics (i.e. acoustical space models or other acoustic characieristics), aud their respective 
comrol parameters, indices, and buffers. The impulse response data is retrieved from the co- 
efScient memory storage device 60 by a load and process routine 110 in response to a user 
command mouitored by a load and process routine 1 10 via an I/O control routine 11 L Routine 
110 is comprised of routines 62 and 64 CFIG 2), Jh particular, the I/O control routine simply 
monitors user inputs to the GUIs illustrated in FIGS. I A or IB and retrieves the data structures of 
the co-efficieuts that correspoud to Uie user selected acoustic characteristic. The load and 
process routine 110 simply loads the selected data structures into the run-time mepiory 100 on a 
channel by channel basis. These data structures are identified in the runtime memory as 
IMPULSE 1, IMPULSE 2,., IMPULSE M, 106, 107 and 108, respectively- As shown in FIG 4, 
the frequency domain data PXLBfl, PXLAfl ; PXLBfZ, PXLAI2;. . .PXLB£N> PXLAfN from the 
frame buffers 102, 103, 104 and the data structures pld, plc2...plcM, 106, 107. 108. 
respectively is communicated to a chaw^el sequencing module 118 which serves to time- 
multiplex the data for processing by the process S3, particular, in&rmation passed from the 
channel sequencing module 118 to the process channel module 120 includes, for each of the K 
audio input channels, data representing a time synchronized frrst framed portion of each frame of 
data received via that audio input channel (PXLBIli), i = I, 2,.-,N), data representing a time- 
synchronized second framed portion of the same data rec^ved via thai audio input channel 
(PXLAf(i), i « 1,2, -.-N)- Other variables are also passed to the process channel module 53; 
Plc(i) i$ a pointer to the tagDynamicChannelData data of impulse channel (i), PlOBufti) is a 
pointer to the output buffer of impulse channel (i), dwFRAMBSize is the number of time domain 
samples input into and ouq)ut from the process channel routine 53 each time it is called by the 



AMENDED SHEET 



PAGE 2IW36 ' RCVD AT 8/512005 3:00:20 PM [Eastern Daylight Time] * SVR:USPTO€FXRF-6/25 * DNIS:27332fl1 * CSID:3129021fl81 ' DURATION (mnKS):0MO 



V' "T / |j j} c": ir-ji ({J.. \n\ p\ r\\ n & r\\ r\\ \t 



•I } 



08-05-05 02:08pn] From-KATTEN MUCHIN ROSENMAN 13129021061 



3129021061 



T-774 P. 21/36 F-363 




host, PI is The pointer- ro ihe instance data struciure, which is unique to the instance but shared 
amongst the pJuraJity of chaiwels for each ixistance, simulated stereo is a control bit which 
enables/disables the simulated stereo function, M/S decode is a control bit which 
enables/disables the Mid-Side audio decoder function, and control is a real time scheduling 
connol bit which enables left and right channels to be processed on separate frames to facilitate 
real time processing CPU load balancing. All of ihis data passes from the channel sequencing 
module 118 to the process channel routine 120, As also shown in FIG 4, bi-directional 
communication is provided between the process channel routine 53 and the run time memory 
100, wdicated by arrows 122. 




[00411 A plurality of T output buffers OUT I, OUT 2,,, OUT T, identified with the reference 
numerals 112. 113, 1 14. are provided in the run-time memory 100. Each of the output buffers 
112, 1 13 and 114 is sized to receive one frame of output audio samples at a tinie for outputting 
the respective T ouiput sample streams. The output buffer pointers pIOBufl, 
pIOBuC2...pIOBufr fcr the user selected audio characteristic of each channel CH- I, CH- 
N of the input audio samples is time multiplexed by the channel sequencing module 118 
to provide independent references to process channel 53, which synthesizes audio output streams 
in real time into the ouq)ut buffers OUT I. OUT 2,.,OUT T, identified vrttb the reference 
numerals 112, 113 and U4. 



[0042] Multiple copies or multiple instances of the same audio processing system 4S can be used 
simultaneously or in lime multiplex. The multiple instances allow for simultaneous processing, 
for example, of different musical instruments. For example, the relative location of each 
instrument in an orchestra relative to a microphone can be simidated. Since such instruments are 
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played simultaneously, multiple copies or msiances of the audio processing system 48 are 
required in ojder to synthesize tbe effects in real time. As such, the channel sequencing module 
118 must provide appropriate references of all of the copies or instances to the process channel 
module 53. As such^ an instance data buffer J, identified with the reference numeral 116, is 
provided in the runtime memory 100 for each instance of the audio processing system 48 being 
employed. 





[0043] to order to provide a clear understanding of tbe audio processing involved in the present 
invention, a time-domain representation of an exemplaiy impulse response input signal is shown 
graphically in FIG 6. As shown, tbe impulse input sigual includes a time wise first pomon 
designated "b") and a continuous, time wise second portion desigpated *'a", and a "tail" portion 
that extends continuously beyond tbe time wise second portion "a". Jn the time domain, the 
impulse input signal may be partitioned into groups of samples. The first portion of the impulse 
input signal (herein after refezred to as the •^b" portion") preferably includes a number of samples 
corresponding to the m«jor frame size for FFT blocks XLBNA2, and the rime wise second 
portion (herein after referred to as the "a" portion") preferably is made up of a number of such 
frames of samples. There is a minor frame size for FFT blocks XI-ENB2, for example, one 
eighth of the m%or block size in the exemplary embodiment. Tbe total number of samples 
making up the aiwiio signal illustrated in FIO 6 is denoted by FTAPS2. A pointer hindex is used 
to designate a relative position within the aggregate collection of samples making up the 
illustrated audio impulse resfponse or it^ut signal. 

10044] There is a uniqw co-eiScient for tjie "a" portion an(J "b" portion, HindexA and HindexB 
respectively. FJG 8 iUustrates the co-efiEficient index sequencing routine, iUustrated in FIG- 5 by 
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ibe Wock 170, and sjiows the coefficient irulex sequencing derived &om XLENA2, XLENB2, 
HindexA, HindexB. and HLENAA, the later of which is scaled to half the 5i2e of XJ-ENA2 
when operating in mibo mode, otherwise is equaj to XLENA2. The HindexA and HindexB 
indices are derived within block S3 and switched by a conm^l signal LPhaseAB to adapt the 



coefficients within the adaptive filter to accommodate the A and lilbll & portions of the 
impidse response. 





10045J FIG 5 depicts a block diagram illustrating in greater detail the operation of the process 
channel routine 53 (FIGS. 2 and 4) as described above, and in panicnlar the pseudo convolution 
processing routine in accordance with the present invention for nse on general pwpose CPUs, 
such as an Intel Pentium 4 piocessor at a greatly reduced processor load* Conventional frequency 
domain convolution is simply a vector multiply of frequency domain multiplicands, followed by 
an inverse Fonrier or Fast Fourier transfonn of the products at a single, nnifomu non-time 
varying, fixed sample rate and block size, resulting in significantly higher computation and 
dvoughput. Conventional convolution does not contain the processes necessary for framing, 
synchroniaing or processing the multirate input audio sipial, the multirate impulse responses, nor 
does it employ an ad^tive filter with time varying coefficients fior a given impulse response. 



[0046] As shown in FIG. 5, dynamic channel data 150, identified in FIG. 4 as "CONTROy, 
from the channel sequencing module 118 is applied to the process channel routine 53, In 
particular, for each copy or iiisiance of the audio processing system 48, the chaimel sequencing 
routine fojmulates a dynamic data strucmre 150 for each channel based upon the user selected 
audio characteristic and dte incoming audio source samples. More particularly, as mentioned 
above> input audio samples are converted to the fi^quency domain and stored in the runtime 
mcmoiy 100. The impulse response coefficients to the various us^ selectable acoustic 
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clwaaeristics are likewise stored in the nmiime memory 100, AJl of this daia is formulated into 1 

I 

a data structure, for example, the exemplary data structure 150, illustrated in FIG, 5. One data I 

t 

smicture 150 is provided for e^U cbamiel of convolution currently being processed in real time I 
and assigned a separate input channel. 

I 

10047] The data structure 150 may include at a plurality of exemplary data fields 152, 154, 156, ; 
158, 160, 162, 164, 166, wd 168. as shown. As shown in FIG- 5, the frequency domain co- 




efficients Hx(F) of a finite impulse response (FIR) filter are used to form die field 154, The field 



152, accessed via specific reference within the structure pointed to by the plcCn) pointer 
identified in FIG,4, may he used to represent two indexes representing as follows: (I) an index 
reference (bindex B) representing that a time wise first portion impulse response input data is 
being processed; (2) an index reference (bindex A) representing that a time wise second portion 
of the impulse response input data is being processed; and (3) additional control data, such as 
runtiine MicLevel, Perspective, DirectLevel, tail extension audio processing control parameters- 
Simulated stereo connrol nmtime parameters, and other audio digital signal processing 




parameters associated with load time or runtime audio processing. These are described fimher in 



the tagPynamicCbannelData data snucture table below. 



[0048] The field 154 (FIG 5) contains the frequency domain filter coefficients f]x(i). which may 
also be in the fomi of acoustic impulse responses, populated with a frequency domain 
represTOtaiion of an acoustic model being simulated in the fojro of a FIR ie,g., a particular 
acousnc space, a particular nricropbone, a particular musical insmmiem body resonance 
characteristic, etc.). This finite FIR is stored in the data snucmre Hx(Q, sized to accommodate 
twice the number of lime domain samples making up the acoustic model IMPSIZE''2), in 
order to accommodate the freqiiency domain representation. 
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10049] The field 156 U ayJi^cally genei:ate4 and contaixis m mtermediate part of tbe producT 
of 8 vecu>r multipUcauoa &om a vector multiplier 172 of the FJR co-efficients pointed to by 4 
Co-^fScient Ind« Sequencing Routine 170 and tbe frequency domain audio source input data 
XLBFt XLAF, illustrated in »be box, identified witb tbe inference numeral 174, for tbe N 
channels, Tbe buffer X13F contains the fUU sample rate, early portion of tbe impulse response 
or FIR filter coefficients in the firequency domain output firom (FIG 3) into field 94> and when 
n4rbo mode is enabled buffer XJ-AF contains the half sample ra|e. later portion of the inqpulse 
response or FIR filter coefficients, in the frequency domain output fipom (FIG 3) into field 96. 
Tbe Cf intermediate product is converted to the time domain by the Inverse Fast Fourier 
Transfonn joutine JFFT 176 and stored in tbe field 158. The time domain data in field 158 ^en, 
balfHIeu, is applied to a Audio Collection and Index Sequencing Routine 178 which along with 
the collection indices data , acolindexA&B, acolindexPrevA&8, in field 160 is used to develop 
the data in fields 162, 164 and 166, as discussed below. 

[OOSO] Hlen represents in tbe time domain tbe equivalent of one frame of frequency domain 
data. balffCen represents in tbe time domain tbe equivalent of one-half fi^e of firequency 
domain data. 

(OOSl| The field 160 contains iiutices lo past and present frames in tbe audio collection buffer 
for tbe B portion of the impulse reis^onse (acolindexprevB and acolindcxB, respectively, and for 
the A portion of tbe impulse response, acolindexprevA and acolindexB, respectively. The field 
162 contains tbe audio collection buffer (acol) 162 corresponding to tbe processing which 
occurs at the fijll sample rate as indicated by the block 178a (FIG. 7) . (an intOTnediate 
accumulative that £aciliT4tes the overl^add or overlap-subtract) and which comprehends fi^e- 
based overlap and modulo addressing. This bui{er (acol) }62 , is tnodu^o addressed as indict 
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by ibe block 192 (FIG- 7) mid is sized of length impulse size (the time-dommn length of tjie 
impulse response) into which successive ftwnes are overlap added or overlap subtracted. 

100521 FIG 9 and FIG ID show more detail on the maintenance of the audio collect and hindex 
indices within the audio collection and index sequencing 178. Prior to a call to the vector 
multiply 172, Hlen is assigned to either XLenA or Xl«n» and acolindex is assigned to 
acolindexA or acolindexB, and the respective impulse coefficient a»d collection buffer indices 
are modulo updated. This is done as to adapt the frequency domain filter coefficients on the fly 
to blocjc processing of multiple portions of an impulse response within the same filter module. 

[0053] As shown in FIG. 10, the coefficieni index hindex, illustrated in FIG. 6. is set depending 
on which part of the waveform shown in FIG, 6 is being processed. As shown in Fig. 10, if the 
early part of the waveform is being processed, identified in FIG. 6 as "b", as determined by dre 
decision block 203, the coefBcieni index hindex is set to 0.lf the later portion of the wavefoma is 
being processed, identified in FIG. 6 as **a"„ as determined by the decision block 205. the 
coefiTiclem index is set to XlenA2. which is the begirming of portion "a". 

I00S4J As sbowr> in (FIG 7), when the vector multiply and IFFT stages are operating at half of 
die full sample rate, as when in turbo mode and when processing the sample fiom portion A 
asdetermined by the decision block 200, the aitdio collection and index sequencing field as 
generated by the audio collection and index sequencing routine 178 (FIG 5) and respective 
collection indices will phase align and overlap-add respective audio frames from the ct field 158 
into the buffe< acolh), audio collect half sample rate field 1 64. Also shown in (FIG 7), when tail 
extension is selected and set and the coeificiem index, hindex, is greater th«n the tail collection 
Umit,as determined by the decision block 1 78b, then the audio collection and index sequencing 
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routine operates as illustwted by the block 1 78c and the audio colieciion a»d index sequencing 
field 178 (FIG 5) and the respective collection indices vyiU phase align and overlap-add or 
overlap-subtract respective audio fiames from the ct field 158 into the bttffer, acolDH, audio 
coUect delay half rate field 166. The acolb and acolDH buffers, 164 and 166, respectively, are 
modulo addressed as indicated by the WocJcs 192(FIG. 7) and are sized to have a length that is 
half the injpulse size plus the number of taps in the 1 :2 upsawple filler field 180, the tap length 
being added to the buffer size in order to feciliiate buffer tail overlap typical of FIR type filters. 

(0055J After 411 b^f sample mc processing is offeei according to appropriaTe phase by collection 
ijwlices and overlap added imo acojh, the 1:2 upsample block field 180 converts the half sample 
rate data into the ftOl sample rate and accwnulates the result into the audio colleci foil sample 
rate buffer fieldl62 . 

(00S61 After all half sample rate tail extension processing is offset according to appropriate 
phase by collection indices and overtap added into acolDH, the tail extension 1 :2 upsample block 
field 182 converts this tail extension half sample rate data into the Ml sample rate and 
accumulates the result into the tail extension audio collect delay full rate buffer, acolO, field 168, 

10057] Tail extension processing is optionally enabled by the user in order to model the very eiul 
portion of an impulse response to mitigate the fact that convolution processing is very 
CPUintensive, More particularly,raiber than spend valuable computation time on portions of an 

M 

impulse response that may be ncaring the point of inaudibiUty or otherwise less significant than 
earlier portions of an in^)ulse response, tail extension modeling employs an algorithmic model at 
a far lower computational load. For example, if an impulse response is 4 seconds in duration, the 
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last second may be moiieled V> save premium convolution processing lime for only The early part 
of the response. 

I00S8J FIG U is an exemplary tail extension model The model illustrated in FIG, U is 
exemplary and includes a two hasic routines; a copy scale routine asmcpyscale 207 and a filter 
routine asmfbkfilt.209, as shown. Other configurations are also within the scope of the 
invention.Tbe audio data written into a read/write buffer acolD 168. As shown in FIG. 5, the tail 
extension processing routine, processes this d^ta in the buffer acolt) and remm3 it to the buffer 
acol as shown in FIG. 5. 




(0059] The later pordon of the convolution processing, for example the third second in our 4 
second impulse example, may be copied into a buffer. acolDH, at the half sample rate, or acolD 
at the full sample rate. The taU extension model, similar » a conventional reverberation 
algorithm, is synchroiuzed and applied to the late response. There are low pass filters for timbre 
matching, volume control for volume matching to the tail level of the acmal impulse, feedback 
and overlap parameters, all of which facilitate a smooth transition from convolution processing 
to algorithm processing. 



[0060] An important aspect of die invention relates to embedding and controlling convolution 
technology widiin a sampler or synthesizer that is a music sampler for a music synthesiaer 
engine and what tins technology will do is add to the description of a virtual musical instrument. 
One example relates to modeling an acoustic piano. In that example, the behavior of the piano 
soundboazd resonate is emulated. In this example, the parameters tbat control the inq?ulse 
response of the piauo souudboard may be saved into a file description which contains both the 
original samples of the individual notes on the piano and control parameters to dynamically scale 
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die convolution perimeters in real-time sucb th»t the behavior of an acoustic piapo soundboard is 
the same as the model veirsion. So, in essence, the system embeds and controls convolution- 
j«laied parameters within a synthesizer engine-tbus embedding that convolution process inside 
title virtual musical instrument processing itself. Typically, a sampler or synthesizer engine 
includes an interpolator which gives you pitch control, a low-frequency oscillator or LFO* and an 
envelope generator. Envelope generators provides dynamic control of amplitude over time 
which are all processing audio which is routed through a convolution process whete now other 
aspects of the control and npdeling of the sound is coming firom the synthesizer engme in 
dynamically conttolling the convolution process. Examples of dynamically controlling the 
convolution process are,conn:ollmg the pre and post convolution level control, damping of audio 
energy from within the convolution buifers for simulating a damping of a piano soundboard as 
when the damper pedal is raised, changing the wet/dry, adding and subnracting various impulse 
responses representing various atnibutes of a sound, and changing the ^^^^^ctivc conn»r. In 
regards to ^'perspective connrol," what lhat is doing is changing the envelope of the impulse 
response in real-time as a musical insttumeni is being played- 9y combining all of these 
processes, physical insnwnents can be modeled with far greater detail and accuracy than before. 



{0061] Various file snuctures can be employed in which the impulse response associated with 
the sound of a musical insorument, the control parameters associated with the impulse responses, 
the digital sound sample representing single or multiple notes of an instrument, control 
parameters for the synthesizer engine filters. LFO, envelope generators, interpolators, and sound 
generatons are stored together into a file structure representation of a musical instrument TTeus 
file strucmre has single or multiple data fields representing each of these characters of the 
synthesized sound, which may be organized in a variety of ways using a variety of file data 
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types. This mysiwl infiiruraent file smicture may include the ambient enviiopmeni, instromem 
body resonance, microphone type, roicropbone placement, or other aiulio character of the 
synthesized sound. An example file stmcmre is as follows: Impulse Response I... Impulse 
Respause(n), Impulse Response I impulse control I ---Impulse Response I impulse control(m). 



Impulse ResponseCn) impulse conurol I —impulse RespQnse(n) impulse controlCm), digital 



sound sample I . . .digital sound sample(p), sampler engine control parameter I . - - sampler engine 
conm)l parameter (q), synthesizer engine control parameter I synthesizer engine control 
parameter (r), pointer to other file 1, -.pointer to other file(n)- Together, these parameters 
repreaent the sound behavior of a musical instrument or soimd texture generator, in which the 
impulse responses and their interactivity within the syntbesi«er engine via user perfonnance data 
are contributing to the sound produced by the instrument model-] 

An exemplary channel data structure is illustrated below. The Channel Sequencing Routine US (FIG- 
4) chooses the particular pointers and controls that are fed into process channelfJim-we need to provide 
better definition here]. Each instance of this data structure represents one dynamic channel data impulse 
block in Runtime Memory 100, Pieccwise Convolution is done (in portions which are combined) by an 
*'overlap-add'* method (technically, overlap-subtract with a downstream phase reversal). 



typedef struct _tagDynamicChannelDaia 



PataXypc 




Description 


Ipp32f 
32-bit floj 
point 


HxpMPSJZE*2]: 


//filter in^)ulse response (FIR Filter), (Frq domain 
representation of the acoustic model being simulated (e.g., 
acoustic space, microphone, musical instrument body 
resonance)) 


Ipp32f 


acollACOlXENGTHl; 


//audio collect (intermediate accumulator for overlap- 
add) — comprehends frame-based overlap and modulo 
addressing 

modulo addressed buffer sized of len^ impulse size (the 
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time-domam length of the impulse response) mio which 
successive irmnes are overlap added 


Ipp32f 


acom[CACOLLENGTH/2) + 
LPFTAPSl; 


//rmbo collect. Half si^e for reduced sample-raie to save 
CPU overhead 


Ipp32f 


acolDH[(ACOI-LENGTH/2) + 
ILPFTAPS]; 


//turbo tail ext Delay collect. Half size for reduced sample- 
rate to save CPU overhead 


Ipp32f 


acolDCACGlXENGTUh 


//tail extension delay 


Ipp32f 


tarMicI-evel; 


//target mic level scale (new setpoint) 


Ipp32f 


delMicLevel; 


//delta mic level scale (recursively calculated transition 
level gradwlated firom one sample to the jiext to smooth the 
transition) 


lDn3''f 


riiTiMicLfivel; 


//runtime mic level scale (origixial being apnlied) 


lpp32f 


nwMicPerspec; 


//runtime mic perspective scale (affects the envelope of the 
impulse response — used for nmtime scahng) appues a 
volume scaling to early part of re^nse that increases or 
decreases volume to create perception of a close or distant 
pexspoctive on the au4io — coirelated to some scape value 
that makes sense for the user interface of the apparent 
distance of the sound source &om the mic. 


lpp32f 


tJirDirectLevel; 


//target Direct Sim stereo level scale 


Ipp32f 


delDiiwiLeveJ; 


//delta Direct Sim stereo level scale 


Ipp32f 


nmDirectl-evel; 


//nmtime Direct Sim stereo level scale 


Ipp32f 
PWOItD 

integer 


hindexA; 


//alignment dummy 

//sub frame frq HpMndexJ - index reference into impulse 
response (the current potion on which a calculation is being 
performed (A portion, which may be at a lower sample 
rate) 


PWORD 




//audio collect buffer index - current index into collect 
buffers used for overlap-add 


OWOJIO 


acoUndexprevA; 


//previous frame collection buffer index - previous index 
value in the collection or accumulation 


DWORD 


outindex; 


//collect buffer output index - index to where data can be 
read from audio collect buffer (second from top above) and 
sent to output buffer 


DWORD 


hindexB; 


//sub frame frq Hpundex*) 


DWORD 


acolindexB; 


//audio collect buffifr index 


DWORD 


acolindexprevB; 


//previous frame collection buffer index 


DWORD 


dmnmvxxxx; 


//alignment dummy (placeholder) 


DWORD 


dlyWiruJex; 


//tail extension delay Write index (delay needed to align 
modeled portion of the impulse response with the acmal 
result of the impulse response) 


DWORD 


dlyRmd^x; 


//tail extension delay Read index (delay needed to align 
modeled portion of the impulse response with the actual 
result of the impulse response) 


DWORD 


lelA; 


//laU extension state variable (&r Tail Exiension Filter) 


DWORD 


te2A; 


//tail extension state variable 
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DWORD 


FcFbk; 


//tail extension state variable 


DWORD 


FcFbk2; 


/Aail extension state variable .^_„ 


DWORD 


FcFbk3: 


//tail extension state variable 


DWORD 


FcFbk4; 


//tail exrension state variable 


DWORD 


FcFbkS; 


//tail extension state variable 


DWORD 


FcFbk6; 


//tail extension state variable 


DWORD 




//1 6 bvte alifinment dummy 


DwrniD 

4-1 vv vJ|\4-# 


fill unt^^^'" 


-»16bvteauLn^t dummy 




API FALL PASS SAMPLESI" 


//tail extensioru all-pass bulTer 




AP2f ALL PASS SAMPLESI: 


//tail extensiotu all^pass bufiTer 




<i^TfSTM STEREO SAMPLES" 


//sim stereo delay buffer (for slow processoxs; simnlates 
Stereo audio by using stereo audio filters and 
complementary comb filters. OPUQNAL 




SStDOrSIM STEREO SAMPL 


//sim stereo, direct delay buffer 


DWORD 


API r 


//AP buffer read index 


DWORD 


AP2 n 


//AP buffer read index 


DWORD 


SSt r; 


//Sim Stereo buffer read index 


DWORD 


API w; 


//AP buffer write index offset 


DWORD 


AP2 w; 


//AP buffer write index ofiset 


DWORD 


tarSSt 


//target Sim Stereo buffer write index ofiset 


Ipp32f 


larSSTWicWi; 


/Aarget Sim stereo depth 


Ipp32f 


delSStWidih; 


//delta Sim stereo depth 


Jpp32f 


nmSStWidth; 


//runtime Sim stereo depth 


DWORD 


delSSt w; 


//4elta Sim Stereo buffer write index ofifeet 


DWORD 


nmSST w; 


//runtime Sim Stereo buffer write index offset 


DWORD 


SStDD w; 


//sim stereo, direct delay write offset 




, py^UU^Ca^ANNELDATA. ♦PDYNAMICCHANNELDATA; 
The foregoing description is for the purpose of teaching those skilled in the an the best mode of 
carrying out the invention and is to be construed as illustrative only. Numerous modifications 
and alternative embodiments of the invention will be apparent to those skilled in the an in view 
of this description, and the details of the disclosed strucmre may be varied substantially without 
dialling from the spirit of die invention. Accordingly, the exclusive use of all modifications 
within the scope of the appended claims is reserved. 



What is claimed and desired to be covered by a Letters Patent follows: 
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